Fix SymbolMappingMsg hd.length on cross-version encode#21
Merged
Conversation
The Databento Live gateway emits SymbolMappingMsg in v1 wire layout (hd.length=20, 80 bytes total) even when the consumer is writing into a v3 file. The encoder dispatched on metadata.version to pick the body layout (v1: 64-byte body; v2+: 160-byte body) but wrote the record's original hd.length unchanged. The on-disk header then disagreed with the body that followed it, mis-aligning the decoder and corrupting every subsequent record in the stream. Re-derive hd.length from the layout actually being serialized (20 for v1, 44 for v2+) before writing the record header. The struct itself carries only parsed values, so the input version doesn't matter; the output is always self-consistent with the file's metadata.version. Added a regression test that constructs a v1-wire SymbolMappingMsg (hd.length=20), writes it into a v3 file alongside a trailing TradeMsg, and verifies both records round-trip cleanly with the symbology record's length rewritten to 44 on disk. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-1.0 minor bump for the SymbolMappingMsg encoder fix: behavior of the encoder on cross-version writes changed (records that previously produced corrupt files now produce correct ones, and the on-disk hd.length of SymbolMappingMsg records is normalized to the file's metadata.version). Downstream consumers should opt in via their [compat] entry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous commit bumped to 0.2.0; the encoder fix is a bug fix without breaking API change, so patch is the right level. Downstream [compat] "0.1" entries pick it up automatically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SymbolMappingMsg.hd.lengthto match the layout it actually serializes (20for v1,44for v2+), instead of passing the input record'shd.lengththrough unchanged.SymbolMappingMsgin v1 wire layout (hd.length=20, 80 bytes) even when the consumer is writing a v3 file. The encoder picked the v2+ body (160 bytes) but wrote the originalhd.length=20header — the decoder then mis-aligned and every record after the mapping was unreadable.SymbolMappingMsg cross-version (v1 wire → v3 file)) that writes a v1-wire mapping followed by aTradeMsginto a v3 file and verifies both round-trip with the mapping'shd.lengthrewritten to44on disk.Context
Downstream report and workaround: tbeason/DatabentoAPI.jl#14 (JSONL sidecar). Once this fix is released, the workaround can be removed there.
Test plan
include("test/runtests.jl")— Phase 8 testset is now 102 passing (was 94). Other 3408 tests unchanged.pkg> test DatabentoAPI) — all 241 pass against this branch via the local-path dep.SymbolMappingMsg(hd.length=20) + trailingTradeMsgwritten to a v3 file, both decode cleanly; mapping's on-diskhd.lengthis44.🤖 Generated with Claude Code